Papers with Decoder architecture
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI. |
| Approach: | They propose a paradigm called KV-Latent to reduce the KV cache footprint and improve inference speed by down-sampling the Key-Value vector dimensions into a latent space. |
| Outcome: | The proposed paradigm reduces the KV Cache footprint and improves inference speed with a small amount of extra training, less than 1% of pre-training takes. |